ci(docs): scheduled doc e2e persona fleet that executes the documented journeys#708
Open
ojongerius wants to merge 3 commits into
Open
ci(docs): scheduled doc e2e persona fleet that executes the documented journeys#708ojongerius wants to merge 3 commits into
ojongerius wants to merge 3 commits into
Conversation
A fleet of persona 'new users' walks the published docs end to end using only the documentation (install -> use -> inspect) and reports drift to a GitHub tracking issue. Verified findings are cross-checked against SDK source. - .claude/agents/doc-e2e-reviewer.md — read-only persona-walkthrough subagent - .claude/doc-e2e/personas.md — 6 adopter journeys (Python, TS, Go, MCP proxy, hook, dashboard) - .github/workflows/doc-e2e.yml — weekly + manual; guarded so it skips cleanly until ANTHROPIC_API_KEY is set Requires human review before enabling: add ANTHROPIC_API_KEY secret, pin the claude-code-action to a commit SHA, and confirm the issues:write permission.
- Rename the subagent doc-e2e-reviewer -> doc-e2e-runner: it now follows the docs and actually runs each step in a throwaway environment (install, daemon, emit, CLI, dashboard), proving the journey works and reporting steps that fail as written. Adds Write/Bash tools; keeps source-verification of factual claims. - personas.md: reframe for execution; runner uses its real OS and flags missing OS coverage. Rename persona liam-python -> theo-python. - workflow: provide Go/Node/pnpm/uv toolchains so runners can install+run; update the orchestration prompt to the execute-and-prove framing.
The runner's shell does not persist env vars between separate commands, so a one-time 'export XDG_DATA_HOME=...' did not reach a later inspection command — making the runner misread a tool's $HOME-fallback default as 'ignores XDG_DATA_HOME' (a false positive against the dashboard, which actually honors it since v0.3.0). Instruct the runner to persist env to a file and re-source it on every command, and to never log a default read from a shell without the env applied. No persona/journey changes.
ojongerius
force-pushed
the
claude/great-pasteur-YgeUW
branch
from
a55d7df to
72d1531
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a scheduled, persona-driven documentation end-to-end audit. A fleet of "new user" personas follows the published docs and actually executes each step in a throwaway environment — install → run the daemon → emit → inspect with the CLI → view the dashboard — using only what the docs say. It reports anything unclear, missing, wrong, or that simply doesn't work when run, to a single GitHub tracking issue. This catches docs that read fine but don't run.
Pieces
.claude/agents/doc-e2e-runner.md— the subagent. Works inmktemp -dwith its ownXDG_DATA_HOME, follows the journey, runs every step, proves milestones with real output, and distinguishes doc bugs from environment limits. Verifies suspected factual errors against source. Never modifies the repo, commits, or opens issues; cleans up after itself..claude/doc-e2e/personas.md— six adopter journeys: Theo (Python), Maya (TypeScript), Raj (Go), Nina (MCP proxy), Omar (hook), Priya (dashboard). Add a persona = add a block..github/workflows/doc-e2e.yml— weekly + manual. Provisions Go/Node/pnpm/uv (the runners install and build per the docs) and a writableXDG_RUNTIME_DIRfor the daemon socket, then runs the fleet and opens/updates a "Docs e2e drift report" issue.Proven, not hypothetical
A live trial of the Theo/Python runner executed the whole journey on Linux (pip install → built the daemon from source →
--init→ started it → emitted withDaemonEmitter→agent-receipts list/show/verify→ dashboardHTTP 200), and surfaced real drift onmain(stale@v0.8.0install pin breakinglist/show, a verify-path ergonomics issue, a missing PATH note). Those doc fixes are in #707.ANTHROPIC_API_KEYrepo secret. Until then a guard step skips the job, so scheduled runs stay green instead of hard-failing.anthropics/claude-code-action@v1to a commit SHA (there's aTODO(review)on the line) to match this repo's pinned-action convention.permissions:(issues: write) — that's the workflow's only write surface; it never edits files or opens PRs.Notes
environment-limitedin headless CI — that's honest, not a failure..claude/files are force-added (the repo already tracks.claude/hooks/*.sh+settings.jsonthe same way despite the blanket ignore).Generated by Claude Code